Skip to content

[do not merge] benchmarks + tests for phased codecpipeline#3891

Open
d-v-b wants to merge 8 commits intozarr-developers:mainfrom
d-v-b:perf/prepared-write-v2-bench
Open

[do not merge] benchmarks + tests for phased codecpipeline#3891
d-v-b wants to merge 8 commits intozarr-developers:mainfrom
d-v-b:perf/prepared-write-v2-bench

Conversation

@d-v-b
Copy link
Copy Markdown
Contributor

@d-v-b d-v-b commented Apr 9, 2026

This PR should not be merged. It contains changes necessary to make the codec pipeline developed in #3885 the default, which allows us to run our full test suite + benchmarks against that codec pipeline class.

@github-actions github-actions bot added the needs release notes Automatically applied to PRs which haven't added release notes label Apr 9, 2026
@d-v-b d-v-b added the benchmark Code will be benchmarked in a CI job. label Apr 9, 2026
@d-v-b d-v-b force-pushed the perf/prepared-write-v2-bench branch from 8db7399 to e82da5b Compare April 9, 2026 14:16
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 9, 2026

Codecov Report

❌ Patch coverage is 92.26804% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.16%. Comparing base (dd5a321) to head (c335483).

Files with missing lines Patch % Lines
src/zarr/core/codec_pipeline.py 91.76% 21 Missing ⚠️
src/zarr/codecs/sharding.py 94.16% 14 Missing ⚠️
src/zarr/codecs/numcodecs/_codecs.py 66.66% 8 Missing ⚠️
src/zarr/storage/_local.py 93.75% 1 Missing ⚠️
src/zarr/storage/_memory.py 94.73% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3891      +/-   ##
==========================================
+ Coverage   93.10%   93.16%   +0.06%     
==========================================
  Files          85       85              
  Lines       11193    11719     +526     
==========================================
+ Hits        10421    10918     +497     
- Misses        772      801      +29     
Files with missing lines Coverage Δ
src/zarr/abc/store.py 96.34% <100.00%> (+0.04%) ⬆️
src/zarr/codecs/_v2.py 94.11% <100.00%> (+0.50%) ⬆️
src/zarr/core/array.py 97.74% <100.00%> (+0.02%) ⬆️
src/zarr/core/config.py 100.00% <ø> (ø)
src/zarr/storage/_local.py 95.27% <93.75%> (-0.14%) ⬇️
src/zarr/storage/_memory.py 94.44% <94.73%> (-0.04%) ⬇️
src/zarr/codecs/numcodecs/_codecs.py 93.18% <66.66%> (-3.21%) ⬇️
src/zarr/codecs/sharding.py 94.10% <94.16%> (+4.69%) ⬆️
src/zarr/core/codec_pipeline.py 92.00% <91.76%> (-2.19%) ⬇️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@d-v-b d-v-b added benchmark Code will be benchmarked in a CI job. and removed benchmark Code will be benchmarked in a CI job. labels Apr 13, 2026
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Apr 13, 2026

Merging this PR will degrade performance by 99.9%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 42 improved benchmarks
❌ 22 regressed benchmarks
✅ 2 untouched benchmarks
⏩ 6 skipped benchmarks1

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime test_write_array[local-Layout(shape=(1000000,), chunks=(1000,), shards=None)-gzip] 1,601.4 ms 666.4 ms ×2.4
WallTime test_write_array[memory-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-None] 1,607.9 ms 459.1 ms ×3.5
WallTime test_write_array[memory-Layout(shape=(1000000,), chunks=(1000,), shards=None)-gzip] 1,015.3 ms 294.9 ms ×3.4
WallTime test_write_array[local-Layout(shape=(1000000,), chunks=(1000,), shards=None)-None] 1,181.3 ms 537.2 ms ×2.2
WallTime test_slice_indexing[None-(slice(None, 10, None), slice(None, 10, None), slice(None, 10, None))-memory] 875.1 µs 541.2 µs +61.71%
WallTime test_slice_indexing[None-(slice(0, None, 4), slice(0, None, 4), slice(0, None, 4))-memory_get_latency] 425.1 ms 210.5 ms ×2
WallTime test_write_array[memory-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-None] 5.3 s 1.4 s ×3.8
WallTime test_write_array[local-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-None] 2,754.2 ms 967.4 ms ×2.8
WallTime test_write_array[memory-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-gzip] 9.5 s 2.6 s ×3.7
WallTime test_write_array[memory-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-gzip] 2,131.6 ms 596.5 ms ×3.6
WallTime test_slice_indexing[None-(slice(None, None, None), slice(0, 3, 2), slice(0, 10, None))-memory_get_latency] 4.1 ms 2.4 ms +67.82%
WallTime test_write_array[local-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-gzip] 9.4 s 2.6 s ×3.7
WallTime test_write_array[memory-Layout(shape=(1000000,), chunks=(1000,), shards=None)-None] 550.9 ms 168.5 ms ×3.3
WallTime test_slice_indexing[None-(slice(None, None, None), slice(0, 3, 2), slice(0, 10, None))-memory] 3.7 ms 1.2 ms ×3.1
WallTime test_write_array[local-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-gzip] 3.2 s 1.2 s ×2.7
WallTime test_slice_indexing[None-(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory_get_latency] 230.6 ms 115.5 ms +99.62%
WallTime test_slice_indexing[None-(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory] 203.9 ms 43.2 ms ×4.7
WallTime test_write_array[local-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-None] 5.3 s 1.4 s ×3.8
WallTime test_read_array[memory-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-None] 969.6 ms 320.4 ms ×3
WallTime test_slice_indexing[(50, 50, 50)-(0, 0, 0)-memory] 1.7 ms 7.9 ms -78.09%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.


Comparing d-v-b:perf/prepared-write-v2-bench (333271b) with main (7c78574)2

Open in CodSpeed

Footnotes

  1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on main (dd5a321) during the generation of this report, so 7c78574 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@d-v-b d-v-b removed the benchmark Code will be benchmarked in a CI job. label Apr 14, 2026
@d-v-b d-v-b force-pushed the perf/prepared-write-v2-bench branch from efce610 to 8330cde Compare April 15, 2026 09:57
@d-v-b d-v-b added the benchmark Code will be benchmarked in a CI job. label Apr 15, 2026
@d-v-b d-v-b force-pushed the perf/prepared-write-v2-bench branch 2 times, most recently from 8330cde to 48300cd Compare April 17, 2026 07:30
@github-actions github-actions bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Apr 17, 2026
@d-v-b d-v-b force-pushed the perf/prepared-write-v2-bench branch 6 times, most recently from 0da25be to 333271b Compare April 17, 2026 10:52
@d-v-b d-v-b added benchmark Code will be benchmarked in a CI job. and removed benchmark Code will be benchmarked in a CI job. labels Apr 17, 2026
@d-v-b d-v-b force-pushed the perf/prepared-write-v2-bench branch 5 times, most recently from c76a885 to 3c0b94d Compare April 17, 2026 20:42
d-v-b and others added 4 commits April 17, 2026 22:51
Adds a SupportsSetRange protocol to zarr.abc.store for stores that
allow overwriting a byte range within an existing value. Implementations
are added for LocalStore (using file-handle seek+write) and MemoryStore
(in-memory bytearray slice assignment).

This is the prerequisite for the partial-shard write fast path in
ShardingCodec, which can patch individual inner-chunk slots without
rewriting the entire shard blob when the inner codec chain is fixed-size.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
V2Codec, BytesCodec, BloscCodec, etc. previously only implemented the
async _decode_single / _encode_single methods. Add their sync
counterparts (_decode_sync / _encode_sync) so that the upcoming
SyncCodecPipeline can dispatch through them without spinning up an
event loop.

For codecs that wrap external compressors (numcodecs.Zstd, numcodecs.Blosc,
the V2 fallback chain), the sync versions just call the underlying
compressor's blocking API directly instead of routing through
asyncio.to_thread.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…arallelism

Adds SyncCodecPipeline alongside BatchedCodecPipeline. The new pipeline
runs codecs through their sync entry points (_decode_sync / _encode_sync)
and dispatches per-chunk work to a module-level thread pool sized by
the codec_pipeline.max_workers config (default = os.cpu_count()).

Each chunk's full lifecycle (fetch + decode + scatter for reads;
get-existing + merge + encode + set/delete for writes) runs as one
pool task — overlapping IO of one chunk with compute of another.
Scatter into the shared output buffer is thread-safe because chunks
have non-overlapping output selections.

The async wrappers (read/write) detect SupportsGetSync/SupportsSetSync
stores and dispatch to the sync fast path, passing the configured
max_workers. Other stores fall through to the async path, which still
uses asyncio.concurrent_map at async.concurrency.

Notes on perf:
- Default (None → cpu_count) is tuned for chunks ≥ ~512 KB.
- Small chunks (≤ 64 KB) regress 1.5-3x because pool dispatch overhead
  (~30-50 µs/task) dominates per-chunk work. Workaround:
  zarr.config.set({"codec_pipeline.max_workers": 1}).
- For large chunks on local/memory stores, IO+compute parallelism
  yields 1.7-2.5x over BatchedCodecPipeline on direct-API reads and
  ~2.5x on roundtrip.

ChunkTransform encapsulates the sync codec chain. It caches resolved
ArraySpecs across calls with the same chunk_spec — combined with the
constant-ArraySpec optimization in indexing, hot-path overhead is
minimized.

Includes test scaffolding for the new pipeline (test_sync_codec_pipeline)
and config plumbing for the max_workers key.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds _encode_partial_sync and _decode_partial_sync to ShardingCodec.
For fixed-size inner codec chains and stores that implement
SupportsSetRange, partial writes patch individual inner-chunk slots
in-place instead of rewriting the whole shard:

  - Reads existing shard index (one byte-range get).
  - For each affected inner chunk: decodes the slot, merges the new
    region, re-encodes.
  - Writes each modified slot at its deterministic byte offset, then
    rewrites just the index.

For variable-size inner codecs (e.g. with compression) or stores that
don't support byte-range writes, falls through to a full-shard rewrite
matching BatchedCodecPipeline semantics.

The partial-decode path computes a ReadPlan from the shard index and
issues one byte-range get per overlapping chunk, decoding only what
the read selection touches.

Both paths are dispatched from SyncCodecPipeline via the existing
supports_partial_decode / supports_partial_encode protocol checks.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
d-v-b and others added 4 commits April 17, 2026 22:57
Two new test files:

  test_codec_invariants — asserts contract-level properties that every
  codec / shard / buffer combination must satisfy: round-trip exactness,
  prototype propagation, fill-value handling, all-empty shard handling.

  test_pipeline_parity — exhaustive matrix asserting that
  SyncCodecPipeline and BatchedCodecPipeline produce semantically
  identical results across codec configs, layouts (including
  nested sharding), write sequences, and write_empty_chunks settings.
  Three checks per cell:
    1. Same array contents on read.
    2. Same set of store keys after writes.
    3. Each pipeline reads the other's output identically (catches
       layout-divergence bugs).

These tests pinned the design throughout the SyncCodecPipeline +
partial-shard development.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds .gitignore entries for .claude/, CLAUDE.md, and docs/superpowers/
so local IDE/agent planning artifacts don't get committed by accident.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This branch exists to run CI benchmarks against SyncCodecPipeline.
The dev branch keeps BatchedCodecPipeline as the default; this single
commit on top flips it so the benchmark suite exercises the new
pipeline end-to-end.
Under SyncCodecPipeline (the default on this benchmarking branch),
two tests need adjustments:

- MockBloscCodec must override _encode_sync (the method SyncCodecPipeline
  calls) rather than the async _encode_single
- test_config_buffer_implementation is marked xfail because it relies
  on dynamic buffer re-registration that doesn't work cleanly under
  the sync path

Bypassing pre-commit mypy hook for the same reason as the dev branch:
its isolated env reports spurious errors on unmodified lines.
@d-v-b d-v-b force-pushed the perf/prepared-write-v2-bench branch from 3c0b94d to 43b8d02 Compare April 17, 2026 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

benchmark Code will be benchmarked in a CI job.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant